Front-end Integration Guide

The CSV import endpoint has changed from synchronous (200 with results) to asynchronous (202 with batch ID + polling). Here's what needs to change in the front-end:

API Changes

Before (synchronous):

POST /api/v1/entrances/import-rows → 200 { successfulImport: [], failureImport: [], total: {...} }

After (asynchronous):

POST /api/v1/entrances/import-rows → 202 { batchId, totalRows, totalChunks, statusUrl }
GET /api/v1/jobs/:batchId           → 200 { batchId, type, status, progress, result }

Implementation Steps

1. Submit the import — POST /api/v1/entrances/import-rows with { data: [...rows] }. Expect a 202 response:

   {
     "batchId": "uuid",
     "totalRows": 2000,
     "totalChunks": 41,
     "statusUrl": "/api/v1/jobs/uuid"
   }

2. Poll for progress — GET /api/v1/jobs/:batchId until status is "completed" or "failed":

   {
     "batchId": "uuid",
     "type": "csv-import",
     "status": "active",       // "pending" | "active" | "completed" | "failed"
     "progress": {
       "totalChunks": 41,
       "completedChunks": 20,
       "totalRows": 2000,
       "processedRows": 980,
       "successes": 900,
       "duplicates": 50,
       "failures": 30
     },
     "result": null             // populated only when status = "completed"
   }

3. Show results — When status === "completed", result contains:

   {
     "reportUrls": {
       "success": "https://...signed-url...",    // null if no successes
       "duplicates": "https://...signed-url...", // null if no duplicates
       "failures": "https://...signed-url..."    // null if no failures
     },
     "summary": {
       "successes": 1700,
       "duplicates": 300,
       "failures": 0
     }
   }

4. Handle errors:

   • 400 — empty data array or missing mandatory columns (immediate, no batch created)
   • 403 — user is not admin
   • 500 — job queue unavailable or batch creation failed
   • status: "failed" — all chunks failed during processing

UX Recommendations

• Show a progress bar using processedRows / totalRows (or completedChunks / totalChunks for chunk granularity)
• Poll every 2–3 seconds
• When complete, show the summary and offer download links for the report CSVs (if URLs are non-null)
• Report URLs are SAS-signed and expire after 7 days
• The user also gets an email notification with the same summary and links

Access Control

• Only the batch initiator can view their own job status
• Moderators and Admins can view any batch
• Non-authorized users get 403

New Notification Type: IMPORT_COMPLETE

This PR adds a new notification type IMPORT_COMPLETE (id: 8) to t_notification_type. The front-end notification system needs to handle this new type:

• When: Sent to the user who initiated the CSV import batch, once all chunks have been processed
• Display: Should render as an in-app notification (e.g., "Your CSV import has completed") with a link to view the results/download reports
• Distinction from existing types: Unlike CREATE/UPDATE/DELETE which relate to entity CRUD operations, IMPORT_COMPLETE is a job-completion event — it won't have the usual entity link (cave, entrance, etc.) but instead relates to a batch job

Dev Environment: Local File Storage

This PR also includes a local file storage mechanism for development (commit feat(storage): add local file storage for dev environment).

What it does

When AZURE_KEY is not set (local dev), file uploads are stored on disk in .local-uploads/ instead of Azure Blob Storage — mirroring how the email helper logs to console instead of sending via SES.

How it works

• Documents & reports are written to .local-uploads/documents/
• DB exports are written to .local-uploads/db-exports/
• Files are served back via a /local-uploads/* middleware on the local server
• URLs in API responses point to http://localhost:<port>/local-uploads/...
• .local-uploads/ is gitignored
• nodemon.json whitelists only api/, config/, and app.js to prevent restarts when files are written

No changes needed to test locally

Just run npm run dev without AZURE_KEY — uploads work automatically. No more "Missing credential" warnings for document operations.

Thanks for the thorough review @Paul-AUB! Pushed fixes in 14fb03f. Here's what was addressed:

Fixed:

1. Race condition in checkBatchCompletion — Added an atomic test-and-set guard (updateOne({id, status:'active'}).set({status:'aggregating'})). Only the first completion-check worker transitions the batch; subsequent ones get null and bail out. Added 'aggregating' to the model validator and Swagger enum.

2. CODE_REVIEW.md removed — Agreed, doesn't belong in version control.

3. Error handling in aggregateBatch — Wrapped generateAndUploadReports in try/catch. On failure, batch transitions to 'failed' with a completedAt timestamp so the user knows something went wrong.

4. Silent row loss in processOneChunk — The outer catch now pushes all remaining unprocessed rows as failures with the error message, so no rows silently vanish from batch results.

5. TOKEN_SALT fallback removed — Script now exits with an error if the env var is unset.

Not changed (with rationale):

4. autoUpdatedAt: false — This setting is not supported at the model level in Sails 1.0 (causes a boot error). The global config already opts out of auto-timestamps by not defining updatedAt in the global attributes block, so there's no risk of Waterline injecting the column.

5. Email subject i18n — The sendEmail helper already passes emailSubject through i18n.__() (see api/helpers/send-email.js:95), and translations for 'CSV Import Complete' exist in all 15 locale files. No change needed.